Method and apparatus for profile score threshold setting and updating

ABSTRACT

A novel approach for filtering documents involves the use of delivery ratio threshold setting technique to set an initial profile score threshold and the use of beta-gamma regulation for dynamic threshold updating. A group of documents is scored pursuant to a user profile. The score for each document is indicative of the relevance of the corresponding document to the user profile. The score can be compared with a profile score threshold to decide if the document should be accepted or rejected. According to one aspect of the invention, the initial threshold is set to a score threshold that approximates an expected ratio of acceptable documents calibrated with respect to a set of reference documents. According to another aspect of the invention, the score threshold can be updated based on the accumulated example documents, user&#39;s relevance judgment, and the user&#39;s utility function. The accumulated example documents are first scored against a profile and a ranked list of scored documents is obtained. Each position at the ranked list corresponds to a candidate score threshold as well as a utility value computed based on the relevance status of the example documents. From these candidate threshold points, an optimal utility threshold and a zero utility threshold are determined. Using the optimal utility threshold and the zero utility threshold, a new utility threshold is calculated by interpolating between estimates of the optimal utility threshold and the zero utility threshold. This new utility threshold is used for subsequent information retrieval and filtering.

FIELD OF THE INVENTION

[0001] The present invention relates to the field of computerizedinformation search and retrieval systems. More specifically, thisinvention relates to a method and apparatus for setting and updating thescore threshold of a user profile.

BACKGROUND OF THE INVENTION

[0002] Given the vast amount of information accessible by computersystems, particularly on distributed databases, more efficient methodsof information retrieval are continually needed. Often the use of searchtools returns a large volume of data, much of which may not be relevantto the user's ultimate needs. The user is forced to parse through largevolumes of information to find ultimately that which is relevant. It istherefore desirable to develop a system whereby a corpus or a dynamicstream of documents is sufficiently filtered such that only relevantinformation is returned to the user.

[0003] Profile-based filtering involves the interaction of a document orgroup of documents with a user profile. A stream of incoming documentsis compared with certain criteria, contained in a user profile, and theneither rejected or ultimately provided to the user. Conceptually, a userprofile (i.e., a binary document classifier) consists of three keyelements: a term vector, inverse document frequency or “IDF” statistics,and a score threshold. The first two elements are used to assign a scoreto the document, and the third is used to make the decision of whetherto accept or reject the document as relevant or not relevant to theuser's search parameters. The process of profiling is distinct fromdatabase searching in that profiling evaluates and selects or rejectsindividual documents as they stream in rather than evaluating alldocuments of a database and then selecting the best scoring ones as intraditional database searching.

[0004] The basic approach to profile-based filtering involves a two-stepprocedure. For each document-profile pair, a relevance score iscomputed. That score is then applied to a profile score threshold tomake the binary decision to accept or reject the document for theprofile. It is important that the profile score threshold be low enoughsuch that it allows sufficient amounts of relevant documents to bereturned to the user. However, if the profile score threshold is set toolow, a large number of documents will be returned, necessitating furtherfiltering. For any user profile, the optimal threshold should representthe best tradeoff between accepting more relevant documents and avoidingaccepting non-relevant documents, where the best tradeoff is determinedby the user's utility preference.

[0005] Setting the profile score threshold can be divided into twoseparate parts: (a) an initial score threshold setting, before there areany relevance judgments from the user, and (b) updating the scorethreshold, at any point when relevance judgments are fed back into thesystem. Updating the profile score threshold adapts the filteringprocess to the user's specific requirements and thus provides a moreeffective means of information retrieval.

[0006] Consequently, in view of the need for more efficient searchingtechniques and filtering methods, a method by which the profile scorethreshold may be initially set and then updated during use is highlydesirable. A properly set profile score threshold enables the user tosearch a group of documents in a comprehensive manner, such that fewerrelevant documents are missed by the user, but likewise may prevent theuser from becoming inundated with a large number of documents.

SUMMARY OF THE INVENTION

[0007] An approach for initially setting the profile score threshold andupdating the profile score threshold during use in a profile-basedfiltering system is described. The initial threshold is set based on anexpected acceptance ratio of documents specified by the user. To set aninitial threshold, a set of reference documents (i.e., a referencedatabase) is selected. Each reference document is scored against theprofile and all the reference documents are sorted by their scores. Theinitial threshold is then set to such a score that the ratio ofreference documents with a score above it and those with a score belowit equals the expected acceptance ratio. When user relevance feedback isavailable, the threshold can be updated based on a specific utilityfunction specified by the user. To update a threshold, first a set ofhistorical example documents is identified for any profile. Each exampledocument is scored against the profile and all the example documents aresorted by their scores. Assuming each example document score as apossible candidate threshold, a utility value can be computed for thecandidate threshold. Using the utilities at each candidate threshold,the point of highest utility and the point of zero utility are thendetermined. An updated utility threshold is then calculated byinterpolating between the threshold at the point of highest utility andthe threshold at the point of zero utility, according to the formulasdisclosed herein. The updated utility threshold is then used forsubsequent information retrieval.

BRIEF DESCRIPTION OF THE DRAWINGS

[0008]FIG. 1 is a flow chart that illustrates a method according to thepresent invention for retrieving relevant information from a corpus or astream of documents.

[0009]FIG. 2 is a graph that illustrates a method according to thepresent invention for setting an initial threshold of a user profile inan information retrieval/filtering system.

[0010]FIG. 3 is a flow chart that illustrates a method according to thepresent invention for updating the utility threshold of a user profilein an information retrieval/filtering system.

[0011]FIG. 4 is a graph that illustrates the parameters of the presentinvention used for updating the utility threshold.

[0012]FIG. 5 is a block diagram of a computer system on whichembodiments of the invention may be implemented.

DESCRIPTION OF THE PREFERRED EMBODIMENT

[0013] The approach for retrieving information in accord with theinvention involves the use of any profile scoring mechanism along withtwo threshold setting methods referred to as delivery ratio thresholdsetting and “beta-gamma” regulation respectively. The profile scoringmechanism assigns a score to any document with respect to the profile.The score represents the relevance of the document to a user criteriadefined by the profile. The delivery ratio threshold setting techniquesets an initial score threshold for a profile by approximating aspecified ratio of documents to be delivered or accepted for a profile.The approximation is based on a set of reference documents thatapproximate the documents which will be processed. The beta-gammaregulation technique selects a new profile threshold θ′ by interpolatingbetween estimates of the “optimal” threshold θ_(opt) and the “zero”threshold θ_(zero) over the relevance judgments and the historicaltraining data the system has accrued at any given point. The updatedprofile score threshold is used in subsequent filtering applications toprovide a more accurate and more efficient method of informationretrieval. Such an updating of threshold may be as frequent as needed.

[0014] The approach for retrieving information using delivery ratiothreshold setting for initial profile score threshold setting and usingbeta-gamma regulation for profile score threshold updating according toan embodiment of the invention is now described in more detail withreference to FIGS. 1-5. FIG. 1 illustrates the general method forretrieving relevant information from a corpus of documents 101.According to one embodiment of the invention, a document is a text filecontaining one or more strings of characters or other symbols that areused to form more complex constructs. For example, strings of charactersmay form words, phrases, sentences, and paragraphs. The constructscontained in the documents are not limited to constructs or formsassociated with any particular language.

[0015] In this embodiment, the user profile 102 incorporates a termvector 103 and a score threshold 104. The term vector 103 is used togenerate a score in step 105 for each document in the corpus ofdocuments 101. The score threshold 104 is used for deciding to accept orreject documents in step 106 with respect to each document based uponthe scoring obtained in step 105. If the score of a document is abovethe score threshold, the document will be accepted, otherwise, it willbe rejected.

[0016] In the preferred embodiment, the corpus of documents 101 isprocessed one document at a time. For each document, noun phrases andindividual words are extracted as indexing terms, so as to obtain a termvector. In an alternative embodiment, the corpus of document 101 can besegmented into small subsets of “chunks” of documents. A chunk ofdocuments can be processed together to increase efficiency.

[0017] The scoring in step 105 is performed using standard statisticalanalysis techniques such as vector space-type scoring. In a vectorspace-type scoring system, a score is generated by comparing thesimilarity between a profile (or query) Q and the document D andevaluating their shared and disjoint terms over an orthogonal space ofall terms. For example, the similarities score can be computed by thefollowing formula:${S\left( {Q_{i},D_{j}} \right)} = {\frac{Q_{i} \cdot D_{j}}{\left| Q \middle| {\cdot |D|} \right.} = \frac{\sum\limits_{k = 1}^{t}\left( {q_{ik} \cdot d_{ik}} \right)}{\sqrt{\sum\limits_{k = 1}^{t}{= q_{ik}^{2}}}\sqrt{\sum\limits_{k = 1}^{t}{= d_{ik}^{2}}}}}$

[0018] where Q_(i) refers to terms in the profile and D_(j) refers toterms in the document. The vector space-type scoring technique can beillustrated on the following sample set of profiles and documents: TermsQ₁ Q₂ D₁ D₂ dog 1 1 2 — cat 1 — — 1 hat 1 — 1 — bat 1 — — — mat 1 1 — —hut — 1 2 — cut — 1 — 2 luck — — 3 — buck — — 1 — muck — — — 3

[0019] In this table, the Terms column lists a union of all the termscontained in the two documents D₁ and D₂. The scores of D₁ and D₂ referto the frequency of those terms as they appear in the documents. Thescores of Q₁ and Q₂ refer to frequency of the terms as they appear inthe query. The similarity score of the query Q₁ to document D₁ iscomputed as:${S_{G}\left( {Q_{1},D_{1}} \right)} = \frac{\left( {1 \cdot 2} \right) + \left( {1 \cdot 1} \right)}{\sqrt{1^{2} + 1^{2} + 1^{2} + 1^{2} + 1^{2}} \cdot \sqrt{2^{2} + 1^{2} + 2^{2} + 3^{2} + 1^{2}}}$

[0020] Also, the similarity of the profile Q₁ to document D₂ is computedas: S_(G)(Q₁, D₁)=0.12.

[0021] As can be seen from the above example, the similarity score ofprofile Q₁ to document D₁ is higher than the similarity score of profileQ₁ to document D₂. As a result, the similarity score provides a relativemeasure of the relevance of a document to the profile. A highly-scoreddocument is more likely to be relevant to a profile than a low-scoringone. Therefore, a high score threshold would only allow a fewhigh-scoring documents to be accepted. Most of these high-scoringdocuments may be expected to be relevant to the profile. On the otherhand, a low score threshold would allow more documents to be accepted.However, the ratio of actually relevant documents among thesedocuments—referred to as precision—may be low. The correct threshold canonly be determined according to the user's actual preference concerningthe number amount of documents accepted as well as the expectedprecision of the accepted documents. FIG. 2 illustrates an embodiment ofthe invention used to set an initial score threshold 104. A set ofreference documents is identified as reference database. The profileterm vector is used to assign a score to each reference document. Thereference documents are sorted by their scores to generate a sorted listof reference documents. The expected delivery ratio provided by the userdetermines a cutoff point at the list. Assuming that the user expects toaccept a fraction r of documents from the corpus of documents(e.g.,10%), the cutoff point will be the k-th document in the ranked list,where K=r×N, and N equals the number of documents in the referencedatabase. The score at the cutoff point is taken as the assignedthreshold. In special cases when K<1 or when K>N, heuristicextrapolation is applied.

[0022] The thresholding operation in step 106 determines whether adocument will be delivered to the user in step 107. Documents yielding ascore from step 105 above the score threshold 104 are accepted asrelevant in step 106 and delivered to the user in step 107. Conversely,documents yielding a score below the score threshold 104 are rejected asnot relevant and discarded.

[0023] In step 108, relevance feedback for each accepted document isthen obtained based upon the user's particular needs. The documents thatthe system has already processed serve as a training corpus for updatingthe user profile 102 in step 110 for the filtering of subsequentdocuments in the corpus of documents 101. This updating of the userprofile 102 in step 110 can be done as frequently as needed. Thefrequency of updating can be determined based on the amount of newdelivered documents or the time elapse since last updating. Optionally,profile editor 109 may be used to update user profile 102 directlywithout regard to the results obtained in step 107.

[0024] In the preferred embodiment, the user profile 102 is updated instep 110 by expanding the term vector 103 and re-estimating, accordingto the present invention, the score threshold 104. To expand the termvector 103, standard Rocchio feedback may be used, where the centroidvector of the relevant document vectors is computed and the terms areranked according to their centroid weight. Preferably, however, the Kbest-ranked terms are assigned a uniform weight before they are mergedinto the current term vector 103. K grows heuristically with the numberof relevant documents N available for training, according to thefunction: K=10+10·log(N+1).

[0025]FIG. 3 illustrates an embodiment of the present invention used toupdate the score threshold 104 in step 110. In step 201, documents froma reference dataset (or initial training set) are scored against theprofile vector, and are sorted according to their scores. At eachposition in the ranked list, a utility value U_(i) can be computed byassuming a threshold that is equal to the score of the document at thatposition. Therefore, each position yields a candidate score thresholdand a corresponding utility value. Thereafter, the “optimal” utilitythreshold θ_(opt) is determined in step 203, and the “zero” utilitythreshold θ_(zero) is determined in step 204. The optimal utilitythreshold θ_(opt) is the threshold that yields the highest utility overthe accumulated training data. The zero utility threshold θ_(zero) isthe highest threshold below the optimal utility threshold θ_(opt) thatgives a non-positive utility over the training data under the assumptionthat all documents that were rejected are non-relevant.

[0026] Using the optimal utility threshold θ_(opt) and the zero utilitythreshold θ_(zero), a new profile utility threshold θ′ is thencalculated in step 205 by interpolating between the empirical optimalutility threshold θ_(opt) and the zero utility threshold θ_(zero) overthe historical training data the system has accrued at any given point.As documents are filtered using this process, they are added to thehistorical training data for the system. In this way, the optimumutility is updated as new documents are evaluated.

[0027] The interpolation between the optimal and zero utility thresholdusing a constant parameter α, may be calculated according to thefollowing evaluation formula:

θ=αθ_(zero)+(1−α)θ_(opt)

[0028] The parameter α can be empirically set to any value between zeroand one (alpha-regulation). In the preferred embodiment, a is expressedas a function of two further parameters, β and γ (beta-gamma function206), as reflected in the following calculation:

α=β+(1−β)Ie^(−γM)

[0029] in which M equals the number of training documents upon which therelevance feedback in step 108 of FIG. 1 is performed. In the preferredembodiment, the new profile utility threshold θ′ replaces the previousscore threshold 104 and is used along with the newly expanded termvector 103 to filter any subsequent documents in steps 105 and 106.

[0030] In writing the parameter α in terms of β and γ in the beta-gammafunction 306, both aspects of the bias present in the optimal utilitythreshold θ_(opt) calculation are captured. First, β represents a scorebias correction factor that compensates for the relatively higher scoresof relevant documents in the training corpus. Second, γ expresses thatthe estimated optimal utility threshold θ_(opt) approximates the trueoptimal utility threshold more closely when more judged trainingexamples are available. The parameter γ is the inverse of the number ofdocuments at which the profile utility threshold is placed atapproximately the midpoint of the range between the optimal utilitythreshold θ_(opt) and the zero utility threshold θ_(zero). If fewer than$\frac{1}{\gamma}$

[0031] training examples are available, the profile utility thresholdwill be somewhat lower. By contrast, if more than $\frac{1}{\gamma}$

[0032] training examples are available, the profile utility thresholdwill be somewhat higher.

[0033]FIG. 4 illustrates how a choice of the parameter α determines acutoff point between the points of optimal and zero utility and how theparameters β and γ help to dynamically adjust parameter α according tothe number M of judged documents in the training database. Given aranked list of all of the documents in the training database sorted bytheir scores, their relevance, and a specific utility criterion, theutility value at each different cutoff position can be plotted. Eachcutoff position corresponds to a utility threshold.

Hardware Overview

[0034]FIG. 5 is a block diagram which illustrates a computer system 300upon which an embodiment of the invention may be implemented. Computersystem 300 includes a bus 302 or other communication mechanism forcommunicating information, and a processor 304 coupled with bus 302 forprocessing information. Computer system 300 also includes a main memory306, such as a random access memory (RAM) or other dynamic storagedevice, coupled to bus 302 for storing information and instructions tobe executed by processor 304. Main memory 306 also may be used forstoring temporary variables or other intermediate information duringexecution of instructions to be executed by processor 304. Computersystem 300 further includes a read only memory (ROM) 308 or other staticstorage device coupled to bus 302 for storing static information andinstructions for processor 304. A storage device 310, such as a magneticdisk or optical disk, is provided and coupled to bus 302 for storinginformation and instructions.

[0035] Computer system 300 may be coupled via bus 302 to a display 312,such as a cathode ray tube (CRT), for displaying information to acomputer user. An input device 314, including alphanumeric and otherkeys, is coupled to bus 302 for communicating information and commandselections to processor 304. Another type of user input device is cursorcontrol 316, such as a mouse, a trackball, or cursor direction keys forcommunicating direction information and command selections to processor304 and for controlling cursor movement on display 312. This inputdevice typically has two degrees of freedom in two axes, a first axis(e.g., x) and a second axis (e.g., y), which allows the device tospecify positions in a plane.

[0036] The invention is related to the use of computer system 300 toretrieving information using beta-gamma regulation of thresholdupdating. According to one embodiment of the invention, retrievinginformation using beta-gamma regulation of threshold updating isprovided by computer system 300 in response to processor 304 executingsequences of instructions contained in main memory 306. Suchinstructions may be read into main memory 306 from anothercomputer-readable medium, such as storage device 310. However, thecomputer-readable medium is not limited to devices such as storagedevice 310. For example, the computer-readable medium may include afloppy disk, a flexible disk, hard disk, magnetic tape, or any othermagnetic medium, a CD-ROM, any other optical medium, a RAM, a PROM, andEPROM, a FLASH-EPROM, any other memory chip or cartridge, or any othermedium from which a computer can read. Execution of the sequences ofinstructions contained in main memory 306 causes processor 304 toperform the process steps previously described. In alternativeembodiments, hard-wired circuitry may be used in place of or incombination with software instructions to implement the invention. Thus,embodiments of the invention are not limited to any specific combinationof hardware circuitry and software.

[0037] Computer system 300 also includes a communication interface 318coupled to bus 302. Communication interface 318 provides a two-way datacommunication coupling to a network link 320 that is connected to alocal network 322. For example, communication interface 318 may be anintegrated services digital network (ISDN) card or a modem to provide adata communication connection to a corresponding type of telephone line.As another example, communication interface 318 may be a local areanetwork (LAN) card to provide a data communication connection to acompatible LAN. Wireless links may also be implemented. In any suchimplementation, communication interface 318 sends and receiveselectrical, electromagnetic or optical signals which carry digital datastreams representing various types of information.

[0038] Network link 320 typically provides data communication throughone or more networks to other data devices. For example, network link320 may provide a connection through local network 322 to a hostcomputer 324 or to data equipment operated by an Internet ServiceProvider (ISP) 326. ISP 326 in turn provides data communication servicesthrough the world wide packet data communication network now commonlyreferred to as the “Internet” 328. Local network 322 and Internet 328both use electrical, electromagnetic or optical signals which carrydigital data streams. The signals through the various networks and thesignals on network link 320 and through communication interface 318,which carry the digital data to and from computer system 300, areexemplary forms of carrier waves transporting the information.

[0039] Computer system 300 can send messages and receive data, includingprogram code, through the network(s), network link 320 and communicationinterface 318. In the Internet 328 for example, a server 330 mighttransmit a requested code for an application program through Internet328, ISP 326, local network 322 and communication interface 318. Inaccordance with the invention, one such downloaded application providesfor the retrieval of information using chunks of text as describedherein.

[0040] The received code may be executed by processor 304 as it isreceived, and/or stored in storage device 310, or other non-volatilestorage for later execution. In this manner, computer system 300 mayobtain application code in the form of a carrier wave.

[0041] To measure the effectiveness of the present invention over simplelinear interpolation (i.e., alpha regulation), 49 user profiles wereused to filter about 250 megabytes of 1988 Associate Press newsarticles. A set of 1987 Wall Street Journal documents were used as aninitial reference database. The initial threshold is set for all theprofiles with a delivery ratio of 0.0005 using the present invention.The utility function used in the evaluation is the utility function UF1defined below:

I01=3*R−2*N

[0042] where, R is the number of relevant documents accepted and N isthe number of non-relevant documents accepted.

[0043] Three experiments were conducted. In one experiment, the initialthreshold was kept without updating. The other two experiments updatethe threshold using the present invention in two different ways—one usesthe alpha regulation and the other uses the beta-gamma regulation.Updating frequency is such that a profile will be updated whenever thereare four new documents accepted for the profile. The UF1 utility valuefor each profile and their average are shown in the following table.baseline α regulation improve α-γ regulation improve Profile# (noupdating) (α = 0.3) α reg. over baseline) (β = 0.1, γ = 0.05) (β-γ reg.over baseline) 1 −14 −8 6 −8 6 2 −15 −12 3 −12 3 3 −3 1 4 −11 −8 4 3 1−2 1 −2 5 0 15 15 27 27 6 −16 −4 12 −4 12 7 9 1 −8 26 17 8 −108 −5 103−5 103 9 −26 1 27 −3 23 10 3 6 3 46 43 11 −106 −4 102 −25 81 12 −31 5 36−3 28 13 −16 −13 3 −10 6 14 −1 3 4 2 3 15 10 −3 −13 −5 −15 16 −29 −2 27−2 27 17 46 32 −14 60 14 18 −475 −24 451 −24 451 19 −4 −6 −2 −6 −2 20 26 4 10 8 21 14 8 −6 35 21 23 73 4 −69 110 37 24 15 −3 −18 −7 −22 25 −68−5 63 −16 52 26 −14 2 16 −4 10 27 −1 3 4 3 4 28 −26 −6 20 −6 20 29 −9 −27 −2 7 30 −2 −2 0 −2 0 31 −3 3 6 −3 0 32 −24 −6 18 −6 18 33 −8 −4 4 −4 434 −10 −10 0 −10 0 35 −24 −8 16 −8 16 36 4 4 0 0 −4 37 −12 −2 10 −2 1038 12 5 −7 1 −11 39 −14 −10 4 −10 4 40 21 −1 −22 14 −7 41 −9 −4 5 −4 542 7 9 2 −2 −9 43 −30 5 35 −9 21 44 2 −8 −10 −8 −10 45 −10 −8 2 −8 2 46−187 −20 167 −21 166 47 0 −1 −1 −1 −1 48 −16 −12 4 −12 4 49 7 3 −4 7 050 −16 −8 8 −8 8 Average −22.42857143 −1.714285714 20.714285711.448979592 23.87755102

[0044] These results show that threshold updating using the presentinvention (both the alpha regulation and the beta-gamma regulation)generally improves the utility values, and in some cases significantly,when compared with the performance without updating. The comparisonbetween the alpha regulation and the beta-gamma regulation indicatesthat the beta-gamma regulation technique gives more stable utilityperformance and is more adaptive when a profile has the potential ofachieving a high positive utility. For example, referring to topic 23,although the alpha regulation gives a very small utility, the beta-gammaregulation generates a very high positive utility.

[0045] While this invention has been particularly described andillustrated with reference to particular embodiments thereof, it will beunderstood by those skilled in the art that changes in the abovedescription or illustrations may be made with respect to form or detailwithout departing from the spirit or scope of the invention.

I claim:
 1. A computer-implemented method for filtering documents,comprising the steps of: selecting a document profile and an expecteddocument delivery ratio; scoring a reference set of documents accordingto said document profile; determining an assigned score thresholdcorresponding to said expected document delivery ratio; determining autility function by calculating a utility for each of said documents insaid reference set; determining a first utility threshold based on saidutility function; determining a second utility threshold based on saidutility function; interpolating between said first utility threshold andsaid second utility threshold to determine an updated score threshold;and filtering incoming documents based on said updated score threshold.2. A method as in claim 1, wherein said interpolation is linear.
 3. Amethod as in claim 1, wherein said first utility threshold (θ_(opt)) isthe highest utility over said reference set and said second utilitythreshold (θ_(zero)) is the highest utility below said first utilitythreshold that has a non-positive utility over said reference set.
 4. Amethod as in claim 3, wherein: said interpolation is calculatedaccording to the formula: θ=αθ_(zero)+(1−α)θ_(opt)
 5. A method as inclaim 4, wherein said parameter is calculated according to the formula:α=β+(1−β)e ^(−γM)
 6. A system for filtering documents, comprising: acomputer coupled to a network wherein said computer receives documentsover said network and transmits documents to an individual user oversaid network, wherein said user selects a document profile and anexpected delivery ratio; and wherein said computer: scores a referenceset of documents according to said document profile; determines anassigned score threshold corresponding to said expected documentdelivery ratio; determines a utility function by calculating a utilityfor each of said documents in said reference set; determines a firstutility threshold based on said utility function; determines a secondutility threshold based on said utility function; interpolates betweensaid first utility threshold and said second utility threshold todetermine an updated score threshold; and filters incoming documentsbased on said updated score threshold.
 7. A system as in claim 6,wherein said interpolation is linear.
 8. A system as in claim 6, whereinsaid first utility threshold (θ_(opt)) is the highest utility over saidreference set and said second utility threshold (θ_(zero)) is thehighest utility below said first utility threshold that has anon-positive utility over said reference set.
 9. A system as in claim 8,wherein: said interpolation is calculated according to the formula:θ=αθ_(zero)+(1−α)θ_(opt)
 10. A system as in claim 9, wherein saidparameter is calculated according to the formula: α=β+(1−β)e ^(−γM).